Papers with informativeness metric
Stay Hungry, Stay Focused: Generating Informative and Specific Questions in Information-Seeking Conversations (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work on question generation assumes knowledge of what the answer might be . instead, questioner must reason pragmatically about how to acquire new information . |
| Approach: | They propose a question generation system that generates pragmatically relevant questions in information-asymmetric conversations. |
| Outcome: | The proposed questioner significantly improves the informativeness and specificity of questions generated over a baseline model as evaluated by metrics as well as humans. |
ReCEval: Evaluating Reasoning Chains via Correctness and Informativeness (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods focus on whether the reasoning chain leads to the correct conclusion, but this view may confound reasoning quality with other spurious shortcuts to predict the answer. |
| Approach: | They propose a framework that evaluates reasoning chains via two key properties: (1) correctness, i.e., each step makes a valid inference based on information contained within the step, preceding steps, and input context, and (2) informativeness, respectively. |
| Outcome: | The proposed framework evaluates reasoning chains via two key properties: (1) correctness, i.e., each step makes a valid inference based on information contained within the step, preceding steps, and input context, and (2) informativeness, which is helpful towards deriving the generated answer. |